{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<i>Copyright (c) Microsoft Corporation. All rights reserved.</i>\n",
    "\n",
    "<i>Licensed under the MIT License.</i>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Testing different Hyperparameters and Benchmarking"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook, we'll cover how to test different hyperparameters for a particular dataset and how to benchmark different parameters across a group of datasets. Note that this re-uses functionality which was already introduced and described in the [classification/notebooks/11_exploring_hyperparameters.ipynb](../../classification/notebooks/11_exploring_hyperparameters.ipynb) notebook. **Please refer to that notebook for all explanations, which this notebook will not repeat.**\n",
    "\n",
    "For an example of how to scale up with remote GPU clusters on Azure Machine Learning, please view [24_exploring_hyperparameters_on_azureml.ipynb](../../classification/notebooks/24_exploring_hyperparameters_on_azureml.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing hyperparameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ensure edits to libraries are loaded and plotting is shown in the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We start by importing the utilities we need."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'1.0.57'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import sys\n",
    "import numpy as np\n",
    "import scrapbook as sb\n",
    "import torch\n",
    "import fastai\n",
    "from fastai.vision import DatasetType\n",
    "\n",
    "sys.path.append(\"../../\")\n",
    "from utils_cv.classification.data import Urls\n",
    "from utils_cv.common.data import unzip_url\n",
    "from utils_cv.classification.parameter_sweeper import ParameterSweeper, clean_sweeper_df, plot_sweeper_df\n",
    "from utils_cv.similarity.data import comparative_set_builder\n",
    "from utils_cv.similarity.metrics import positive_image_ranks\n",
    "from utils_cv.similarity.model import compute_features_learner\n",
    "\n",
    "fastai.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define the datasets and parameters we will use in this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "tags": [
     "parameters"
    ]
   },
   "outputs": [],
   "source": [
    "DATA_PATHS = [unzip_url(Urls.fridge_objects_path, exist_ok=True), unzip_url(Urls.fridge_objects_watermark_path, exist_ok=True)]\n",
    "REPS = 3\n",
    "LEARNING_RATES = [1e-3, 1e-4, 1e-5]\n",
    "IM_SIZES = [300, 500]\n",
    "EPOCHS = [16]\n",
    "DROPOUTS = [0]  #Leave dropout at zero. Higher values tend to perform significantly worse"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Similiarity accuracy metric\n",
    "\n",
    "For image classification, we used the percentage of correctly labeled images to measure accuracy. For image retrieval, our measure is the rank of the positive example among a large number of negatives. This was described in the [01_training_and_evaluation_introduction.ipynb](01_training_and_evaluation_introduction.ipynb) notebook, and we will re-use some of the code from that notebook in the definition of the _retrieval_rank()_ function below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def retrieval_rank(learn):\n",
    "    data = learn.data\n",
    "\n",
    "    # Build multiple sets of comparative images from the validation images\n",
    "    comparative_sets = comparative_set_builder(\n",
    "        data.valid_ds, num_sets=1000, num_negatives=99\n",
    "    )\n",
    "\n",
    "    # Use penultimate layer as image representation\n",
    "    embedding_layer = learn.model[1][-2]\n",
    "        \n",
    "    # Compute DNN features for all validation images\n",
    "    valid_features = compute_features_learner(\n",
    "        data, DatasetType.Valid, learn, embedding_layer\n",
    "    )\n",
    "    assert len(list(valid_features.values())[0]) == 512\n",
    "\n",
    "    # For each comparative set compute the distances between the query image and all reference images\n",
    "    for cs in comparative_sets:\n",
    "        cs.compute_distances(valid_features)\n",
    "\n",
    "    # Compute the median rank of the positive example over all comparative sets\n",
    "    ranks = positive_image_ranks(comparative_sets)\n",
    "    median_rank = np.median(ranks)\n",
    "    return median_rank"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using Python <a name=\"python\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We start by creating the Parameter Sweeper object. Before we start testing, it's a good idea to see what the default parameters are. We can use a the property `parameters` to easily see those default values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "OrderedDict([('learning_rate', [0.0001]),\n",
       "             ('epochs', [15]),\n",
       "             ('batch_size', [16]),\n",
       "             ('im_size', [299]),\n",
       "             ('architecture',\n",
       "              [<Architecture.resnet18: functools.partial(<function resnet18 at 0x000001B6A1648798>)>]),\n",
       "             ('transform', [True]),\n",
       "             ('dropout', [0.5]),\n",
       "             ('weight_decay', [0.01]),\n",
       "             ('training_schedule',\n",
       "              [<TrainingSchedule.head_first_then_body: 'head_first_then_body'>]),\n",
       "             ('discriminative_lr', [False]),\n",
       "             ('one_cycle_policy', [True])])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sweeper = ParameterSweeper(metric_name=\"rank\")\n",
    "sweeper.parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we know the defaults, we can pass it the parameters we want to test, and run the parameter sweep."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running 1 of 6 permutations. Repeat 1 of 3.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "                background: #F44336;\n",
       "            }\n",
       "        </style>\n",
       "      <progress value='1' class='' max='4' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      25.00% [1/4 00:13<00:39]\n",
       "    </div>\n",
       "    \n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1.835002</td>\n",
       "      <td>1.948479</td>\n",
       "      <td>0.431818</td>\n",
       "      <td>00:13</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>\n",
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "                background: #F44336;\n",
       "            }\n",
       "        </style>\n",
       "      <progress value='2' class='' max='5' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      40.00% [2/5 00:01<00:01 1.5208]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "ERROR - Internal Python error in the inspect module.\n",
      "Below is the traceback from this internal error.\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Traceback (most recent call last):\n",
      "  File \"C:\\Miniconda\\envs\\cv\\lib\\site-packages\\IPython\\core\\interactiveshell.py\", line 3331, in run_code\n",
      "    exec(code_obj, self.user_global_ns, self.user_ns)\n",
      "  File \"<ipython-input-6-f6864b7f403f>\", line 2, in <module>\n",
      "    df = sweeper.run(datasets=DATA_PATHS, reps=REPS, metric_fct=retrieval_rank);\n",
      "  File \"../..\\utils_cv\\classification\\parameter_sweeper.py\", line 526, in run\n",
      "    dataset, permutation, early_stopping\n",
      "  File \"../..\\utils_cv\\classification\\parameter_sweeper.py\", line 456, in _learn\n",
      "    fit(learn, head_epochs, head_learning_rate, weight_decay)()\n",
      "  File \"C:\\Miniconda\\envs\\cv\\lib\\site-packages\\fastai\\train.py\", line 22, in fit_one_cycle\n",
      "    learn.fit(cyc_len, max_lr, wd=wd, callbacks=callbacks)\n",
      "  File \"C:\\Miniconda\\envs\\cv\\lib\\site-packages\\fastai\\basic_train.py\", line 202, in fit\n",
      "    fit(epochs, self, metrics=self.metrics, callbacks=self.callbacks+callbacks)\n",
      "  File \"C:\\Miniconda\\envs\\cv\\lib\\site-packages\\fastai\\basic_train.py\", line 99, in fit\n",
      "    for xb,yb in progress_bar(learn.data.train_dl, parent=pbar):\n",
      "  File \"C:\\Miniconda\\envs\\cv\\lib\\site-packages\\fastprogress\\fastprogress.py\", line 41, in __iter__\n",
      "    for i,o in enumerate(self.gen):\n",
      "  File \"C:\\Miniconda\\envs\\cv\\lib\\site-packages\\fastai\\basic_data.py\", line 75, in __iter__\n",
      "    for b in self.dl: yield self.proc_batch(b)\n",
      "  File \"C:\\Miniconda\\envs\\cv\\lib\\site-packages\\torch\\utils\\data\\dataloader.py\", line 346, in __next__\n",
      "    data = self.dataset_fetcher.fetch(index)  # may raise StopIteration\n",
      "  File \"C:\\Miniconda\\envs\\cv\\lib\\site-packages\\torch\\utils\\data\\_utils\\fetch.py\", line 44, in fetch\n",
      "    data = [self.dataset[idx] for idx in possibly_batched_index]\n",
      "  File \"C:\\Miniconda\\envs\\cv\\lib\\site-packages\\torch\\utils\\data\\_utils\\fetch.py\", line 44, in <listcomp>\n",
      "    data = [self.dataset[idx] for idx in possibly_batched_index]\n",
      "  File \"C:\\Miniconda\\envs\\cv\\lib\\site-packages\\fastai\\data_block.py\", line 651, in __getitem__\n",
      "    x = x.apply_tfms(self.tfms, **self.tfmargs)\n",
      "  File \"C:\\Miniconda\\envs\\cv\\lib\\site-packages\\fastai\\vision\\image.py\", line 123, in apply_tfms\n",
      "    else: x = tfm(x)\n",
      "  File \"C:\\Miniconda\\envs\\cv\\lib\\site-packages\\fastai\\vision\\image.py\", line 518, in __call__\n",
      "    return self.tfm(x, *args, **{**self.resolved, **kwargs}) if self.do_run else x\n",
      "  File \"C:\\Miniconda\\envs\\cv\\lib\\site-packages\\fastai\\vision\\image.py\", line 464, in __call__\n",
      "    if args: return self.calc(*args, **kwargs)\n",
      "  File \"C:\\Miniconda\\envs\\cv\\lib\\site-packages\\fastai\\vision\\image.py\", line 469, in calc\n",
      "    if self._wrap: return getattr(x, self._wrap)(self.func, *args, **kwargs)\n",
      "  File \"C:\\Miniconda\\envs\\cv\\lib\\site-packages\\fastai\\vision\\image.py\", line 167, in lighting\n",
      "    self.logit_px = func(self.logit_px, *args, **kwargs)\n",
      "  File \"C:\\Miniconda\\envs\\cv\\lib\\site-packages\\fastai\\vision\\image.py\", line 206, in logit_px\n",
      "    if self._logit_px is None: self._logit_px = logit_(self.px)\n",
      "  File \"C:\\Miniconda\\envs\\cv\\lib\\site-packages\\fastai\\vision\\image.py\", line 145, in px\n",
      "    self.refresh()\n",
      "  File \"C:\\Miniconda\\envs\\cv\\lib\\site-packages\\fastai\\vision\\image.py\", line 132, in refresh\n",
      "    self._px = _grid_sample(self._px, self.flow, **self.sample_kwargs)\n",
      "  File \"C:\\Miniconda\\envs\\cv\\lib\\site-packages\\fastai\\vision\\image.py\", line 535, in _grid_sample\n",
      "    return F.grid_sample(x[None], coords, mode=mode, padding_mode=padding_mode)[0]\n",
      "  File \"C:\\Miniconda\\envs\\cv\\lib\\site-packages\\torch\\nn\\functional.py\", line 2656, in grid_sample\n",
      "    return torch.grid_sampler(input, grid, mode_enum, padding_mode_enum)\n",
      "KeyboardInterrupt\n",
      "\n",
      "During handling of the above exception, another exception occurred:\n",
      "\n",
      "Traceback (most recent call last):\n",
      "  File \"C:\\Miniconda\\envs\\cv\\lib\\site-packages\\IPython\\core\\interactiveshell.py\", line 2044, in showtraceback\n",
      "    stb = value._render_traceback_()\n",
      "AttributeError: 'KeyboardInterrupt' object has no attribute '_render_traceback_'\n",
      "\n",
      "During handling of the above exception, another exception occurred:\n",
      "\n",
      "Traceback (most recent call last):\n",
      "  File \"C:\\Miniconda\\envs\\cv\\lib\\site-packages\\IPython\\core\\ultratb.py\", line 1148, in get_records\n",
      "    return _fixed_getinnerframes(etb, number_of_lines_of_context, tb_offset)\n",
      "  File \"C:\\Miniconda\\envs\\cv\\lib\\site-packages\\IPython\\core\\ultratb.py\", line 316, in wrapped\n",
      "    return f(*args, **kwargs)\n",
      "  File \"C:\\Miniconda\\envs\\cv\\lib\\site-packages\\IPython\\core\\ultratb.py\", line 350, in _fixed_getinnerframes\n",
      "    records = fix_frame_records_filenames(inspect.getinnerframes(etb, context))\n",
      "  File \"C:\\Miniconda\\envs\\cv\\lib\\inspect.py\", line 1502, in getinnerframes\n",
      "    frameinfo = (tb.tb_frame,) + getframeinfo(tb, context)\n",
      "  File \"C:\\Miniconda\\envs\\cv\\lib\\inspect.py\", line 1460, in getframeinfo\n",
      "    filename = getsourcefile(frame) or getfile(frame)\n",
      "  File \"C:\\Miniconda\\envs\\cv\\lib\\inspect.py\", line 696, in getsourcefile\n",
      "    if getattr(getmodule(object, filename), '__loader__', None) is not None:\n",
      "  File \"C:\\Miniconda\\envs\\cv\\lib\\inspect.py\", line 739, in getmodule\n",
      "    f = getabsfile(module)\n",
      "  File \"C:\\Miniconda\\envs\\cv\\lib\\inspect.py\", line 708, in getabsfile\n",
      "    _filename = getsourcefile(object) or getfile(object)\n",
      "  File \"C:\\Miniconda\\envs\\cv\\lib\\inspect.py\", line 693, in getsourcefile\n",
      "    if os.path.exists(filename):\n",
      "  File \"C:\\Miniconda\\envs\\cv\\lib\\genericpath.py\", line 19, in exists\n",
      "    os.stat(path)\n",
      "KeyboardInterrupt\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m"
     ]
    }
   ],
   "source": [
    "sweeper.update_parameters(learning_rate=LEARNING_RATES, im_size=IM_SIZES, epochs=EPOCHS, dropout=DROPOUTS)\n",
    "df = sweeper.run(datasets=DATA_PATHS, reps=REPS, metric_fct=retrieval_rank); \n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualize Results <a name=\"visualize\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we read in multi-index dataframe, index 0 represents the run number, index 1 represents a single permutation of parameters, and index 2 represents the dataset. To see the results, show the df using the `clean_sweeper_df` helper function. This will display all the hyperparameters in a nice, readable way."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = clean_sweeper_df(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we've run our benchmarking over 3 repetitions, we may want to just look at the averages across the different __run numbers__."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.mean(level=(1,2)).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print the average accuracy over the different runs for each dataset independently."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ax = df.mean(level=(1,2))[\"rank\"].unstack().plot(kind='bar', figsize=(12, 6))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Additionally, we may want simply to see which set of hyperparameters perform the best across the different __datasets__. We can do that by averaging the results of the different datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "df.mean(level=(1)).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To make it easier to see which permutation did the best, we can plot the results using the `plot_sweeper_df` helper function. This plot will help us easily see which parameters offer the highest accuracies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_sweeper_df(df.mean(level=(1)), sort_by=\"rank\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Preserve some of the notebook outputs\n",
    "sb.glue(\"nr_elements\", len(df))\n",
    "sb.glue(\"ranks\", list(df.mean(level=(1))[\"rank\"]))\n",
    "sb.glue(\"max_duration\", df.max().duration)\n",
    "sb.glue(\"min_duration\", df.min().duration)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python (cv)",
   "language": "python",
   "name": "cv"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
